Neural Information Processing Systems http://nips.cc/

In fact, a good damping parameter can be inexpensively estimated from the current iterate.

Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to conic programs converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new $\textit{quadratic growth}$ and $\textit{error bound}$ properties for primal and dual conic programs under the standard strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to conic optimization.

We introduce a limited memory version, L-FCFW, of the Fully-Corrective Frank-Wolfe ( FCFW) method with approximate correction, to solve the dual problem.

Rebuttal for "Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients" Provide a more fine-grained analysis. We hope the reviewer will appreciate the merits of this analysis. Analysis does not apply to neural networks. Convergence analysis of nonconvex nonsmooth objectives is important, and is included in our future research agenda. A discussion with these references will be added.

The structure of this section is as follows: Appendix A.1 describes the notations used in the proof; Appendix A.2 introduces the properties of mixing matrix We use upper case, bold letters for matrices and lower case, bold letters for vectors. The algebraic multiplicity of eigenvalue 1 of W is 1. Thus the algebraic multiplicity of 1 is 1.Theorem II (Perron-Frobenius Theorem for W). The mixing W of RelaySGD satisfies 1. (Positivity) ρ (W) = 1 is an eigenvalue of W . 2. (Simplicity) The algebraic multiplicity of 1 is 1. 3. (Dominance) ρ( W) = | λ Statements 1 and 4 follow from Lemma 4. Statement 2 follows from Lemma 6. Statement 3 follows from Lemma 5 and Lemma 6.Lemma 7 (Gelfand's formula) . We characterize the convergence rate of the consensus distance in the following key lemma: Lemma' 1 Then, we apply Gelfand's formula (Lemma 7) with Lemma 8. Given I in Definition G, we have the following estimate null1 π This assumption is used in the proof of Proposition III. The complete proofs for each case are then given in the following Appendix A.4, The next lemma explains their relations.